Added YUM implementation of OS package build tool by fheinecke · Pull Request #14203 · gravitational/teleport

fheinecke · 2022-07-07T18:43:12Z

This PR has several inter-dependent changes:

Renamed the "build-apt-repos" to "build-os-package-repos"
Rewrote the tool to use subcommands (i.e. from go run . <args> to go run . <apt/yum> <args>)
Implemented YUM repo building per rfd/0058-package-distribution.md
Added os-specific "*.repo" files for use with dnf config-manager and yum-config-manager
Added redirects on APT and YUM buckets from index.html to https://goteleport.com/docs/installation/
Refactored dronegen to more easily support adding OS package repos to the promotion pipeline
Added YUM repo building to dronegen
Added dronegen parallelism support
Added dronegen resource limit support

Before this PR is merged I need to run migrations for old Teleport versions to populate the prod bucket with old release artifacts. Additionally much of it's functionality is dependent on https://github.com/gravitational/cloud-terraform/pull/701 being merged first.

This PR is ready for code review but not yet ready for merge.

github-actions · 2022-07-07T18:43:43Z

@fheinecke - this PR is large and will require admin approval to merge. Consider breaking it up into a series smaller changes.

github-actions · 2022-07-07T18:57:38Z

@fheinecke - this PR is large and will require admin approval to merge. Consider breaking it up into a series smaller changes.

github-actions · 2022-07-07T19:09:07Z

@fheinecke - this PR is large and will require admin approval to merge. Consider breaking it up into a series smaller changes.

github-actions · 2022-07-07T19:39:07Z

@fheinecke - this PR is large and will require admin approval to merge. Consider breaking it up into a series smaller changes.

github-actions · 2022-07-07T19:47:06Z

@fheinecke - this PR is large and will require admin approval to merge. Consider breaking it up into a series smaller changes.

github-actions · 2022-07-08T11:58:43Z

@fheinecke - this PR is large and will require admin approval to merge. Consider breaking it up into a series smaller changes.

r0mant · 2022-07-11T21:09:03Z

@capnspacehook @timothyb89 Could you please take a look at this PR?

timothyb89

Looks good to me, though with the caveat that I haven't really tested anything. Just a couple small questions!

timothyb89 · 2022-07-19T23:02:13Z

+//  * RPM migrations after new YUM pipeline is done
+func artifactMigrationPipeline() []pipeline {
+	migrationVersions := []string{
+		// These versions were migrated as a part of the new `promoteAptPipeline`


This seems a bit unclear to me given all the entries are commented out. Is it a "add the version temporarily to the list then comment it out once migrated" situation? Is there a procedure we could link to if it comes up in the future? (or does it matter?)

That's exactly it. I can write up some docs somewhere... but outside of adding a new OS package repo type (like YUM or APK) it's pretty unlikely that we'll need to use this unless a customer complains about an old version being missing. It's really nice to have (greatly simplifies migrations) but will probably be very infrequently used after this PR is completed.

github-actions · 2022-07-21T20:47:37Z

@fheinecke - this PR is large and will require admin approval to merge. Consider breaking it up into a series smaller changes.

github-actions · 2022-07-21T21:52:23Z

@fheinecke - this PR is large and will require admin approval to merge. Consider breaking it up into a series smaller changes.

github-actions · 2022-07-21T21:54:35Z

@fheinecke - this PR is large and will require admin approval to merge. Consider breaking it up into a series smaller changes.

github-actions · 2022-07-21T22:16:51Z

@fheinecke - this PR is large and will require admin approval to merge. Consider breaking it up into a series smaller changes.

…mplementation

github-actions · 2022-07-27T16:27:01Z

@fheinecke - this PR is large and will require admin approval to merge. Consider breaking it up into a series smaller changes.

github-actions · 2022-07-27T17:13:46Z

@fheinecke - this PR is large and will require admin approval to merge. Consider breaking it up into a series smaller changes.

github-actions · 2022-07-29T00:09:14Z

@fheinecke - this PR is large and will require admin approval to merge. Consider breaking it up into a series smaller changes.

github-actions · 2022-07-29T00:16:47Z

@fheinecke - this PR is large and will require admin approval to merge. Consider breaking it up into a series smaller changes.

…mplementation

* Added YUM implementation of OS package build tool * Addressed PR comments * Added YUM migrations * Added curl to YUM dependencies * Changed pipelines to use golang:1.18.4-bullseye for Go * Implemented proper repo downloading logic * Fixed other merge conflicts * Added artifacts cleanup * Removed delete on s3 sync * Added RPM migrations * v8 migrations * Partial v8 migration * Migration remainder * Reduced requested resources * Updated resource limits per step * Added k8s stage resource limits to drone * Fixed format issue * Removed resource requests * Added `depends_on` support to dronegen * v8.3 migrations * Fixed parallelism * Removed migration parallelism * Fixed RPM base arch lookup * v6 and v7 YUM migration * Fixed missing ISA * Updated repo file path * Added logging * Removed vars from repo file * v8.3 migration first batch * v8.3 migration second batch * v9.0 migration * v9.1 migration * v9.2 migration * v9.3 first migration * v9.3 second migration * v10.0 migration * Removed migrations * Disabled shell linting non-issues * Fixed linter problem * More linter fixes

… into branch/v8 (#15350) Added YUM implementation of OS package build tool (#14203) * Added YUM implementation of OS package build tool * Addressed PR comments * Added YUM migrations * Added curl to YUM dependencies * Changed pipelines to use golang:1.18.4-bullseye for Go * Implemented proper repo downloading logic * Fixed other merge conflicts * Added artifacts cleanup * Removed delete on s3 sync * Added RPM migrations * v8 migrations * Partial v8 migration * Migration remainder * Reduced requested resources * Updated resource limits per step * Added k8s stage resource limits to drone * Fixed format issue * Removed resource requests * Added `depends_on` support to dronegen * v8.3 migrations * Fixed parallelism * Removed migration parallelism * Fixed RPM base arch lookup * v6 and v7 YUM migration * Fixed missing ISA * Updated repo file path * Added logging * Removed vars from repo file * v8.3 migration first batch * v8.3 migration second batch * v9.0 migration * v9.1 migration * v9.2 migration * v9.3 first migration * v9.3 second migration * v10.0 migration * Removed migrations * Disabled shell linting non-issues * Fixed linter problem * More linter fixes

…tional#14203) into branch/v9 (gravitational#15317) * Added YUM implementation of OS package build tool (gravitational#14203) * Added YUM implementation of OS package build tool * Addressed PR comments * Added YUM migrations * Added curl to YUM dependencies * Changed pipelines to use golang:1.18.4-bullseye for Go * Implemented proper repo downloading logic * Fixed other merge conflicts * Added artifacts cleanup * Removed delete on s3 sync * Added RPM migrations * v8 migrations * Partial v8 migration * Migration remainder * Reduced requested resources * Updated resource limits per step * Added k8s stage resource limits to drone * Fixed format issue * Removed resource requests * Added `depends_on` support to dronegen * v8.3 migrations * Fixed parallelism * Removed migration parallelism * Fixed RPM base arch lookup * v6 and v7 YUM migration * Fixed missing ISA * Updated repo file path * Added logging * Removed vars from repo file * v8.3 migration first batch * v8.3 migration second batch * v9.0 migration * v9.1 migration * v9.2 migration * v9.3 first migration * v9.3 second migration * v10.0 migration * Removed migrations * Disabled shell linting non-issues * Fixed linter problem * More linter fixes * Fixed merge issue with dropped function

… into branch/v10 (#15127) Added YUM implementation of OS package build tool (#14203) * Added YUM implementation of OS package build tool * Addressed PR comments * Added YUM migrations * Added curl to YUM dependencies * Changed pipelines to use golang:1.18.4-bullseye for Go * Implemented proper repo downloading logic * Fixed other merge conflicts * Added artifacts cleanup * Removed delete on s3 sync * Added RPM migrations * v8 migrations * Partial v8 migration * Migration remainder * Reduced requested resources * Updated resource limits per step * Added k8s stage resource limits to drone * Fixed format issue * Removed resource requests * Added `depends_on` support to dronegen * v8.3 migrations * Fixed parallelism * Removed migration parallelism * Fixed RPM base arch lookup * v6 and v7 YUM migration * Fixed missing ISA * Updated repo file path * Added logging * Removed vars from repo file * v8.3 migration first batch * v8.3 migration second batch * v9.0 migration * v9.1 migration * v9.2 migration * v9.3 first migration * v9.3 second migration * v10.0 migration * Removed migrations * Disabled shell linting non-issues * Fixed linter problem * More linter fixes

github-actions Bot requested review from capnspacehook and timothyb89 July 7, 2022 18:43

r0mant requested a review from tcsc July 7, 2022 18:53

Added YUM implementation of OS package build tool

60e1ea2

fheinecke force-pushed the fred/rfd-0058-YUM-implementation branch from 3622a14 to 60e1ea2 Compare July 7, 2022 19:46

tcsc reviewed Jul 8, 2022

View reviewed changes

Comment thread build.assets/tooling/cmd/build-os-package-repos/gpg.go

capnspacehook approved these changes Jul 19, 2022

View reviewed changes

Comment thread dronegen/apt.go Outdated

Comment thread build.assets/tooling/cmd/build-os-package-repos/command_executor.go Outdated

timothyb89 approved these changes Jul 19, 2022

View reviewed changes

fheinecke mentioned this pull request Jul 21, 2022

Drone publishing for Amazon ECR #14347

Merged

Addressed PR comments

63d5f1c

Added YUM migrations

4b8c71e

Added curl to YUM dependencies

511d07d

Changed pipelines to use golang:1.18.4-bullseye for Go

7fe7de3

fheinecke added 3 commits July 27, 2022 11:10

Implemented proper repo downloading logic

ac7174b

Merge remote-tracking branch 'origin/master' into fred/rfd-0058-YUM-i…

fca6f10

…mplementation

Fixed other merge conflicts

ec32dda

Added artifacts cleanup

6e9561f

Fixed linter problem

a02ef87

fheinecke enabled auto-merge (squash) July 29, 2022 00:10

More linter fixes

306ee13

fheinecke disabled auto-merge August 2, 2022 16:38

fheinecke added 2 commits August 2, 2022 11:44

Merge remote-tracking branch 'origin/master' into fred/rfd-0058-YUM-i…

7e11834

…mplementation

Resigned dronegen

54288e6

fheinecke enabled auto-merge (squash) August 2, 2022 16:49

Merge branch 'master' into fred/rfd-0058-YUM-implementation

0c4d340

This was referenced Aug 2, 2022

Backport "Added YUM implementation of OS package build tool" (#14203) into branch/v10 #15127

Merged

Backport "Added YUM implementation of OS package build tool" (#14203) into branch/v9 #15129

Closed

Merge branch 'master' into fred/rfd-0058-YUM-implementation

305efc0

fheinecke merged commit b022fea into master Aug 2, 2022

fheinecke added backport backport-required and removed backport labels Aug 3, 2022

fheinecke mentioned this pull request Aug 5, 2022

Added docs for new RPM repos #15268

Merged

fheinecke mentioned this pull request Aug 8, 2022

Backport "Added YUM implementation of OS package build tool" (#14203) into branch/v9 #15317

Merged

fheinecke mentioned this pull request Aug 9, 2022

Backport "Added YUM implementation of OS package build tool" (#14203) into branch/v8 #15350

Merged

This was referenced Aug 15, 2022

Backport #15268: Added docs for new RPM repos (#15268) #15533

Merged

Backport #15268: Added docs for new RPM repos #15534

Merged

Backport #15268: Added docs for new RPM repos #15535

Merged

zmb3 deleted the fred/rfd-0058-YUM-implementation branch September 9, 2022 18:53

Conversation

fheinecke commented Jul 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 7, 2022

Uh oh!

github-actions Bot commented Jul 7, 2022

Uh oh!

github-actions Bot commented Jul 7, 2022

Uh oh!

github-actions Bot commented Jul 7, 2022

Uh oh!

github-actions Bot commented Jul 7, 2022

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2022

Uh oh!

r0mant commented Jul 11, 2022

Uh oh!

Uh oh!

Uh oh!

timothyb89 left a comment

Choose a reason for hiding this comment

Uh oh!

timothyb89 Jul 19, 2022

Choose a reason for hiding this comment

Uh oh!

fheinecke Jul 20, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 21, 2022

Uh oh!

github-actions Bot commented Jul 21, 2022

Uh oh!

github-actions Bot commented Jul 21, 2022

Uh oh!

github-actions Bot commented Jul 21, 2022

Uh oh!

github-actions Bot commented Jul 27, 2022

Uh oh!

github-actions Bot commented Jul 27, 2022

Uh oh!

github-actions Bot commented Jul 29, 2022

Uh oh!

github-actions Bot commented Jul 29, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

fheinecke commented Jul 7, 2022 •

edited

Loading